Kernel-Kernel communication in a shared-memory multiprocessor

In the standard kernel organization on a bus-based multiprocessor, all processors share the code and data of the operating system; explicit synchronization is used to control access to kernel data structures. Distributed-memory multicomputers use an alternative approach, in which each instance of the kernel performs local operations directly and uses remote invocation to perform remote operations. Either approach to inter-kernel communication can be used in a large-scale shared-memory multiprocessor. In this paper we discuss the issues and architectural features that must be considered when choosing between remote memory access and remote invocation. We focus in particular on experience with the Psyche multiprocessor operating system on the BBN Butterfly Plus. We find that the Butterfly architecture is biased towards the use of remote invocation for kernel operations that perform a significant number of memory references, and that current architectural trends are likely to increase this bias in future machines. This conclusion suggests that straightforward parallelization of existing kernels (e.g. by using semaphores to protect shared data) is unlikely in the future to yield acceptable performance. We note, however, that remote memory access is useful for small, frequently-executed operations, and is likely to remain so. hhhhhhhhhhhhhhhhhhhhhhhhhhhhh This research was supported by NSF grant no. CCR-9005633, NSF Institutional Infrastructure grant no. CDA-8822724, a DARPA/NASA Graduate Research Assistantship in Parallel Processing, the Federal University of Rio de Janeiro, and the Brazilian National Research Council. Eliseu Chaves is with the Universidade Federal do Rio de Janeiro, Brazil. He spent six months on leave at the University of Rochester in 1990. Prakash Das is now with Transarc Corp. in Pittsburgh, PA. Brian Marsh is now with the Matsushita Information Technology Lab in Princeton, NJ. Current electronic addresses are: [email protected] (Chaves), [email protected], [email protected], and {leblanc,scott}@cs.rochester.edu.